Image processing apparatus, imaging apparatus, and control method

ABSTRACT

An imaging apparatus performs control to display first region information indicating a first region that is a region gazed at by a user and second region information indicating a second region that is a region of a main object within a region of a shot image region so as to be distinguishably superimposed on the shot image. In the above control, the imaging apparatus displays the first region information and the second region information based on the information on the identical position before the first instruction means is operated, and after the first instruction means is operated, the first region information and the second region information are displayed based on information of different positions corresponding to each region.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image processing apparatus, animaging apparatus, and a control method.

Description of the Related Art

As a result of the progress of the improvement in functionality ofimaging apparatuses, an imaging apparatus having a function that furtherenables an operation in response to a user's intuition, for example, afunction that provides instructions for a focusing position by using atouch panel, has been proposed. Additionally, an interface utilizing aresult for the detection of a gaze region that is a region being gazedat by the user has conventionally been proposed.

Japanese Patent Application Laid-Open No. 2015-22207 discloses atechnique that performs focusing in response to the intention by a userby setting a priority of each detection unit with reference to thesetting of a camera (referred to as a “mode”), in addition to the resultof the detection of the gaze region and the result of the detection ofan object. Additionally, Japanese Patent Application Laid-Open No.5-232374 discloses a method of determining a final object position bydetermining the reliability of the gaze region detection in an objecttracking system using the detection of the gaze region and photometricmeans.

In neither of the techniques in Japanese Patent Application Laid-OpenNo. 2015-22207 nor Japanese Patent Application Laid-Open No. 5-232374are the region of the main object for performing focusing and the gazeregion displayed in a separately recognizable manner. Here, if an erroris cased in the determination of the result of the detection of theobject or the result of the photometry, a tracking frame and the likeare displayed on the region of the object not intended by the user, andas a result, the user has an unnatural feeling. In contrast, if both theregion of the main object and the gaze region are always recognizablydisplayed before the shooting field angle and the main object aredetermined, the display may be confusing and visually undesirable forthe user.

SUMMARY OF THE INVENTION

The present invention improves the usability for this display byimproving the method of displaying a region of a main object and a gazeregion.

An image processing apparatus according to an embodiment of the presentinvention comprises: circuitry which obtains a shot image; detects afirst region that is a region gazed at by a user within a region of theshot image; determines a second region that is a region of a main objectwithin the region of the shot image; and performs control to displayfirst region information indicating the first region and second regioninformation indicating the second region so as to be distinguishablysuperimposed on the shot image; wherein, before the first instructionmember is operated, the circuitry displays the first region informationand the second region information based on information on the identicalposition, and after the first instruction member is operated, thecircuitry displays the first region information and the second regioninformation based on information on different positions corresponding toeach region.

Further features of the present invention will be described from thefollowing description of the example description with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B illustrate a configuration of an image processingapparatus according to the present embodiment.

FIGS. 2A and 2B illustrate an example of an appearance of an imagingapparatus.

FIG. 3 is a flowchart illustrating an example of an operation process ofthe imaging apparatus.

FIGS. 4A to 4C illustrate an example of shot images displayed by theimage processing apparatus.

FIGS. 5A to 5C illustrate another example of the shot images displayedby the image processing apparatus.

FIGS. 6A to 6C illustrate an example of the shot images if shooting hasbeen performed by using the gaze region tracking mode.

DESCRIPTION OF THE EMBODIMENTS

FIGS. 1A and 1B illustrate a configuration of an image processingapparatus according to the present embodiment. In FIGS. 1A and 1B, animaging apparatus such as a digital camera will be described as anexample of image processing apparatuses. Note that the present inventionis applicable to any image processing apparatus that performs an imageprocess upon receipt of the input of a shot image based on signalsoutput from the imaging element. FIG. 1A illustrates a centralcross-sectional view of the imaging apparatus. FIG. 1B illustrates afunctional block diagram of the imaging apparatus. Components to whichthe same reference numerals are provided in FIGS. 1A and 1B correspondto each other.

A lens unit 2 shown in FIGS. 1A and 1B is a lens device that isattachable to and detachable from a camera body 1. The lens unit 2 andthe camera body 1 are electrically connected at an electrical contact11. The lens unit 2 has a lens system control circuit 12, a lens drivecircuit 13, and a shooting optical system 3. The shooting optical system3 has a plurality of lenses. The lens system control circuit 12 controlsthe entire lens unit 2. An optical axis 4 is the optical axis of theshooting optical system. The lens drive circuit 13 drives a focus lens,a blur correction lens, a diaphragm, and the like.

As shown in FIG. 1B, the camera body 1 includes a camera system controlcircuit 5, an imaging element 6, an image processing circuit 7, a memory8, a display device 9, an operation detection circuit 10, a shuttermechanism 14, and a gaze region detection circuit 15. Additionally, asshown in FIG. 1A, the imaging apparatus includes a rear surface displaydevice 9 a and an electronic viewfinder (EVF) 9 b. The operationdetection circuit 10 includes a release button (not illustrated) andoperates as an instruction means of the present invention. The imageprocessing circuit 7 performs an image process on the signals outputfrom the imaging element 6. Consequently, it is possible to detect anobject in the shot image.

The camera system control circuit 5, the lens system control circuit 12,and the lens drive circuit 13 function as an image state adjustmentcircuit that adjusts the state of the output image presented on theelectronic viewfinder 9 b by changing the state of the shooting opticalsystem 3. The image state adjustment circuit, for example, executes afocus adjustment process and focuses on a predetermined region. Inaddition to the shot image, information useful for the user isindependently displayed on the electronic viewfinder 9 b or the shotimage and the information are displayed superimposed on the electronicviewfinder 9 b. In accordance with the operation of the operationdetection circuit 10, the camera system control circuit 5 determines aselection region (second region) to be used for the control of the imagestate adjustment circuit in the region of the shot image. The selectionregion is, for example, a region of a main object. The selection regionmay be a focus detection region where focusing is performed.Additionally, the gaze region detecting circuit 15 has a projector 15 a,a light separator 15 b, and a gaze region detecting sensor 15 c shown inFIG. 1A. Since the detection of the gaze region is a well-knowntechnique, the detailed description will be omitted. A gaze region isdetected by projecting light rays that do not disturb a human visualfield, such as infrared rays, to a cornea and analyzing a reflectedlight. Specifically, the gaze region detection circuit 15 detects afirst region (gaze region), which is a region gazed at by the user, inthe region of the shot image. The camera system control circuit 5 is adisplay control circuit that performs control to display a trackingmarker and a gaze region marker on the display device 9 in adistinguishably superimposed manner. The tracking marker is secondregion information indicating the selection region (second region). Thegaze region marker is first region information indicating the gazeregion (first region).

The obtaining of shot images will be described. A light (object light)from the object forms an image on an imaging surface of the imagingelement 6 via the shooting optical system 3. The imaging element 6photoelectrically converts the object light. A focus evaluation valueand an appropriate exposure amount can be obtained from the imagingelement 6. Hence, by appropriately adjusting the shooting optical system3 based on this signal, the object light with an appropriate lightamount is exposed to the imaging element 6 and an object image is formedin the vicinity of the imaging element 6.

The image processing circuit 7 internally has an A/D converter, a whitebalance adjustment circuit, a gamma correction circuit, an interpolationcalculation circuit, and the like, and can generate an image forrecording. Additionally, a color interpolation processing circuit isprovided in the image processing circuit 7 and performs a colorinterpolation (demosaicing) process based on a signal of Bayer array togenerate a color image. Additionally, the image processing circuit 7compresses, for example, images, moving images, and voices by using apredetermined method. Furthermore, the image processing circuit 7 cangenerate a blur detection signal based on a comparison between aplurality of images obtained from the imaging element 6. The imagegenerated by the image processing circuit 7 is output to the recordingunit of the memory 8 by the camera system control circuit 5 anddisplayed on the display device 9.

Additionally, the camera system control circuit 5 generates and outputstiming signals and the like during imaging, thereby controlling each ofan imaging system, an image processing system, and a recording andreproducing system in response to an external operation. For example,the operation detecting circuit 10 detects the pressing of a shutterrelease button (not illustrated) so that the drive of the imagingelement 6, the operation of the image processing circuit 7, thecompression process, and the like are controlled. Furthermore, thecamera system control circuit 5 controls the state of each segment ofthe information display apparatus that displays information on, forexample, a liquid crystal monitor by the display device 9. Note that, inthis example, the back face display device 9 a is a touch panel and isconnected to the operation detection circuit 10.

A description will be given of the adjustment operation of the opticalsystem by a control system. The image processing circuit 7 is connectedto the camera system control circuit 5, in which an appropriate focalposition and a diaphragm position are determined based on signals fromthe imaging element 6. The camera system control circuit 5 issues acommand to the lens system control circuit 12 via the electric contact11, and the lens system control circuit 12 appropriately controls thelens drive circuit 13. Moreover, in a mode in which camera shakecorrection is performed, the blur correction lens is appropriatelycontrolled via the lens drive circuit 13 based on signals obtained fromthe imaging element. The lens system control and the lens drive circuitfunction as an optical system drive circuit of the present invention.Additionally, the camera system control circuit 5 controls the shuttermechanism 14 and executes exposure control.

FIGS. 2A and 2B illustrate an example of an appearance of the imagingapparatus. FIG. 2A illustrates the front of the imaging apparatus. FIG.2B illustrates the back of the imaging apparatus. A release button 10 aforms a part of the operation detection circuit 10. Additionally, anoperation button 10 b is an operation button different from the releasebutton. The operation button 10 b also forms a part of the operationdetection circuit 10. The operation button 10 b operates as a triggerinput button.

FIG. 3 is a flowchart illustrating an example of an operation process ofthe imaging apparatus. In S100, when a power source of the imagingapparatus is turned on, the operation starts. In S110, the camera systemcontrol circuit 5 reads out the operation mode. The camera systemcontrol circuit 5 reads out setting S120 that has been predetermined andreflects the setting in the operation. In this example, the operationmode is a mode with respect to an operation in which focusing isperformed. As a first operation mode, there is an object tracking modeindicating that the setting of the position for focusing after focusingis once performed by tracking the main object by image recognition, inother words, focusing is performed to serve the region of the mainobject as the selection region. Additionally, as a second operationmode, there is a gaze region tracking mode indicating the setting of theselection region in which focusing is performed based on the detectedgaze region. Note that, here, although an explanation will be given byusing an example of the operation mode in which a process that focuseson the selection region is performed, the present invention is notlimited thereto. It is also possible to perform processes such as theadjustment of exposure values, the adjustment of colors such as hue andsaturation, the correction of contrast and gradation, or thedetermination of a compression level of image signals, based on theselection region. Additionally, instead of a process during shooting,information indicating the selection region may be recorded togetherwith a shot image to serve as information for reference in an imageprocess during reproduction.

In the camera body 1, an instruction for shooting preparation isprovided by the half-pressing operation of the release button 10 areferred to as “S1”, and focusing is executed. Subsequently, the imageis obtained (shot) by the complete-pressing operation of the releasebutton 10 a referred to as “S2”. The user determines a composition(framing) at the stage before focusing, determines an object by S1,determines an image to be shot (aiming), and performs shooting by S2(shooting). At this time, the operation of determining the focusingposition after aiming is also performed.

Next, in S130, the camera system control circuit 5 displays the trackingmarker and the gaze region marker at the identical position in the shotimage. That is, since there is no concept of tracking before theoperation of S1, the camera system control circuit 5 displays the gazeregion marker and the tracking marker based on the information at thesame position. After the operation of S1, the gaze region marker and thetracking marker are displayed based on information about positions thatdiffer from each other. More specifically, after the operation of S1,the camera system control circuit 5 displays the gaze region markerbased on the information indicating the position of the gaze region, anddisplays the tracking marker based on the information indicating theposition of the selection region. Subsequently, in S140, it isdetermined whether or not S1 has been operated. If S1 has been operatedby pressing the release button 10 a in FIG. 2A halfway, the processproceeds to S150. If S1 has not been operated, the process proceeds toS160.

In S150, the image state adjustment circuit, that is, the camera systemcontrol circuit 5, the lens system control circuit 12, and the lensdrive circuit 13 perform focusing. Further, the image state adjustmentcircuit also obtains a template in the region where focusing has beenperformed. This template is used for template matching to be describedbelow as necessary. That is, the image state adjustment circuitdetermines the selection region to be used for executing an adjustmentprocess of the image state (for example, focusing) in response to theoperation of the first instruction member S1. The first instructionmember is, for example, an operating button for instructions forshooting preparation. The image state adjustment circuit determines, forexample, the selection region within a face region obtained by facedetection. Then, the image state adjustment circuit performs focusing onthe selection region.

In S160, the camera system control circuit 5 instructs the gaze regiondetection circuit 15 to detect the gaze region. Subsequently, in S170,the camera system control circuit 5 displays the current information,that is, the shot image, the tracking marker, and the gaze region markeron the display device 9, and then the process returns to step S140. Thatis, the camera system control circuit 5 displays the tracking marker andthe gaze region marker based on the information indicating the positionof the gaze region.

In the processes following S180, the camera system control circuit 5determines the selection region based on the operation mode with respectto the focusing operation and information indicating a shooting state.The information indicating the shooting state is, for example, a statein which a trigger in S210, to be described below, is detected,information indicating the reliability of the result of the detection ofthe object to be used in S230, and information indicating thereliability of the gaze region in S200. The reliability of the result ofthe detection of the object indicates the magnitude of an evaluationvalue related to the detection of the object. The reliability of thegaze region indicates the magnitude of the evaluation value with respectto the detection of the gaze region. In S180, the camera system controlcircuit 5 determines whether or not the operation mode read out in S110is the object tracking mode. If the operation mode is the objecttracking mode, the process proceeds to S190. If the operation mode isnot the object tracking mode but the gaze region tracking mode, theprocess proceeds to S200.

Next, in S190, the camera system control circuit 5 performs objecttracking by image recognition. Specifically, the camera system controlcircuit 5 tracks the object based on the object detected by imagerecognition. That is, the camera system control circuit 5 determines aregion corresponding to the position of the region of the object thathas been determined based on the shot image to serve as the selectionregion. Since there are many prior documents regarding methods fortracking an object such as template matching, the details will not bedescribed here. At this time, the reliability of the calculation can bemeasured based on of a matching degree of template matching. Themeasured reliability is defined as the reliability of the result of thedetection of the object. Specifically, if the position of the object ina screen has changed, the camera system control circuit 5 movesinformation indicating the position to be focused (for example, a focusdetection frame showing a focus detection region) to follow the object,thereby continuing to focus on the same object.

Next, in S210, the camera system control circuit 5 determines whether ornot a trigger has been detected. If the user operates the operationbutton 10 b, which is a second instruction member different from therelease button 10 a shown in FIG. 3, a trigger is generated. If atrigger has been detected, the process proceeds to S220. If a triggerhas not been detected, the process proceeds to S230.

In S220, the camera system control circuit 5 determines the regioncorresponding to the position of the gaze region to serve as theselection region, and starts tracking again. Specifically, the camerasystem control circuit 5 sets a focus detection frame at the position ofthe gaze region, executes a focus detection process, performs focusing,obtains the template at the position, and starts tracking again. Theprocess of S220 corresponds to the setting of the selection region tothe position of the gaze region once, in other words, corresponds to thedisplay of the tracking marker based on the information indicating theposition of the gaze region. After that, the selection region isdetermined based on the detected object by the process of S190, and thenthe process proceeds to S260. The focus detection frame can be returnedto the gaze region if an unintended transfer in the focus detectionregion is caused due to the process of S210 and the process of S220.

Next, in S230, the camera system control circuit 5 determines whether ornot the user has lost the object (has lost sight of the object). If thereliability of the result of the detection of the object is lower thanthe predetermined threshold value, the camera system control circuit 5determines that the object has been lost. If the reliability of theresult of the detection of the object is equal to or greater than thepredetermined threshold value, the camera system control circuit 5determines that the object has not been lost. If another object crossesthe front of the object and the template matching is veryunsatisfactory, the camera system control circuit 5 determines that theobject has been lost. If the object has been lost, the process proceedsto S220. If the object has not been lost, the process proceeds to S260.If the object has been lost, the focus detection frame can be returnedto the gaze region by the process of S230 and the process of S220.

Additionally, in S200, the camera system control circuit 5 determineswhether or not the reliability of the gaze region by the gaze regiondetection circuit is equal to or greater than the threshold value. Ifthe reliability of the gaze region is equal to or greater than thethreshold value, the process proceeds to S240. If the reliability of thegaze region is lower than the threshold value, the process proceeds toS250. The detection of the gaze region may temporarily become inaccuratedue to the influence of an external light and the like. Thedetermination process of S200 is a process for performing an appropriateoperation in this case.

In S240, the camera system control circuit 5 determines the position ofthe focus detection frame based on the result of the detection of thegaze region by the gaze region detection circuit. Specifically, thecamera system control circuit 5 determines a region at a positioncorresponding to (identical to) the position of the gaze region to serveas the selection region. Then, the camera system control circuit 5tracks the gaze region as the position of the focus detection frame andperforms focusing. Subsequently, the process proceeds to S260.Additionally, in S250, the camera system control circuit 5 performsobject tracking by image recognition. That is, the camera system controlcircuit 5 determines the region corresponding to the position of thegaze region to serve as the selection region. Then, the process proceedsto S260.

Next, in S260, the camera system control circuit 5 displays the currentinformation by the display device. Specifically, a first markercorresponding to the gaze region and a second marker corresponding tothe selection region are displayed superimposed on the shot image.Subsequently, the camera system control circuit 5 determines whether ornot S2 has been operated. If the user presses the release button 10 acompletely to operate S2, the process proceeds to S280. If S2 has notbeen operated, the process proceeds to S290.

In S280, the camera system control circuit 5 obtains the shot imageoutput by shooting, and then, the process proceeds to S290. In S290, thecamera system control circuit 5 determines whether or not S1 is beingmaintained. If S1 is being maintained, the process proceeds to S180. IfS1 is released, the process proceeds to S130. Note that the camerasystem control circuit 5 determines that S1 is being maintained while S2is being operated. That is, if S2 continues to be pressed by, forexample, continuous shooting, the process proceeds with S1 beingmaintained. If the power source is turned off, an interruption occursregardless of the process shown in the flowchart of FIG. 3, and theoperation ends at that point in time.

According to the explanation with reference to FIG. 3, in the gazeregion tracking mode, focusing corresponding to the gaze region isperformed irrespective of S1, which is the operation of the instructionmember by the user. In contrast, in the object tracking mode, theselection between the region corresponding to the detected object (afterS1) and the region corresponding to the gaze region (before S1) isswitched in conjunction with the movement of the instruction member. Byproviding such a plurality of modes, shooting in response to theintention by the user can be performed. Specifically, it is possible toproperly use a mode in a case in which the accurate tracking of the gazeregion is requested and a mode in a case in which focusing is maintainedwhile also confirming a surrounding situation after focusing.

Additionally, paying attention to the state of display, in the objecttracking mode, the first marker and the second marker are displayed atthe identical position before the instruction member is operated (beforeS1). After the instruction member is operated (after S1), the firstmarker and the second marker are displayed based on the valuesassociated with each other. Also in the gaze region tracking mode, thedisplay similar to that in the object tracking mode is performed if itis assumed that the selection region is associated with the gaze region.

FIGS. 4A to 4C illustrate one example of shot images displayed by theimage processing apparatus according to the present embodiment. FIGS.4A, 4B and 4C illustrate three temporally different images in which timehas passed in this order. FIG. 4A is a display example in a state beforeS1 (referred to as “S0”). FIG. 4B is a display example at the momentwhen S1 is pressed. FIG. 4C is a display example in a state in which S1is being maintained.

In FIGS. 4A to 4C, a presented image 20 shows an image (shot image)presented to the EVF 9 b. The first object 21 shows a first object. Agaze region marker 22 shows a gaze region marker (shown by a rectangularframe), serving as the first marker corresponding to the gaze region. Atracking marker 23 shows a tracking marker (shown by “+”) serving as thesecond marker corresponding to the selection region. A second object 24is a second object.

In the examples shown in FIGS. 4A to 4C, the camera system controlcircuit 5 performs display on which the markers are superimposed on theshot image. FIG. 4A illustrates a point in time before S1, wherein theprocess stays in S140 in FIG. 3. In this state, the gaze region marker22 and the tracking marker 23 are displayed so as to be superimposed atthe position of the gaze region by the operations in S160 and S170. Thatis, the gaze region marker 22 and the tracking marker 23 are displayedat the position gazed at by the user.

In FIG. 4B, the user provides instructions for S1 by pressing therelease button 10 a halfway while viewing the object. FIG. 4Bcorresponds to steps from step S140 to step S150 in FIG. 3. In FIG. 4B,the gaze region marker 22 and the tracking marker 23 are displayedsuperimposed, the focusing is performed at the position of the gazeregion, and the template is obtained. In FIG. 4B, focusing is performedon the face portion of the first object 21 and the template is obtainedat that portion.

In FIG. 4C, since S1 is being maintained after S1, the camera systemcontrol circuit 5 executes the operation in the object tracking mode.Since the operation mode is the object tracking mode, processes fromstep S180 to step S190 in FIG. 3 are performed. At this time, in stepS190, the focus detection frame moves by image recognition. That is, ifthe object moves within the screen, the camera system control circuit 5keeps on focusing at a region where the focusing has been performed atthe point in time of S1, and displays the second marker corresponding tothe selection region on the position. In FIG. 4C, the tracking marker 23(second marker) is displayed on the face (selection region) of the firstobject 21. At this time, since the gaze region marker 22 is displayed atthe position corresponding to the detected result of the gaze region,the gaze region marker 22 is not necessarily displayed at the positionthat is the same as that of the tracking marker 23.

In the example of FIG. 4C, the user is paying attention to the secondobject 24. Accordingly, the displays shown in FIGS. 4A to 4C can be usedin, for example, shooting of a sports scene by a user, in which there isa plurality of moving objects (players), wherein focusing is performedon a player serving as an object, and then another player serving asanother object is also searched while staying in focus. In a case wherea ball is passed in a ball game, the switching between the objectsbecomes easy. This is because, after the detection of the trigger inS210 in FIG. 3, the process proceeds to S220, and the focusing andtracking can start again at the gaze region position. If a trigger isgenerated by the operation of the operation button 10 b by the user atthe timing shown in FIG. 4C, the camera system control circuit 5performs focusing on the second object 24, and thereafter tracks thesecond object 24. In another trigger generation method, S1 is releasedonce and S1 is pressed again. By performing this operation, the processproceeds as shown in S290, S130, S140, and S150 in FIG. 3, and thefocusing and tracking can start again at the gaze region position.

FIGS. 5A to 5C illustrate another example of the shot images displayedby the image processing apparatus of the present embodiment. The exampleshown in FIGS. 5A to 5C assumes a case in which the first object 31 isthe main object and the second object 32 is crossing the front of thefirst object when the user is waiting for the best timing for pressingthe shutter. As a scene similar to the example shown in FIGS. 5A to 5C,a school play scene is assumed in which a person crosses the front of aperson serving as the main object. FIGS. 5A, 5B, and 5C illustratetemporally different three images in which time has passed in thisorder. In FIGS. 5A to 5C, the same reference numerals are provided forelements having functions that are the same as those in FIGS. 4A to 4C.In FIGS. 5A to 5C, the first object 31 shows the first object and thesecond object 32 shows the second object. FIG. 5A illustrates the stateat the moment of pressing S1. FIG. 5B and FIG. 5C illustrate the statein which S1 is being maintained.

In the states shown in FIGS. 5A to 5B, the camera system control circuit5 sets the focus detection frame on the first object 31 and performsfocusing. At this time, if the user continues to gaze at the firstobject, the gaze region marker 22 and the tracking marker 23 aredisplayed at substantially the same place. If the user gazes at ananother region, the gaze region marker 22 is displayed at the positionof the detected gaze region and the tracking marker 23 is displayed onthe first object 31, as shown in FIGS. 4A to 4C although not as shown inFIGS. 5A to 5C.

As shown in FIG. 5B, a case is assumed in which the second objectcrosses the front of the main object, thereby causing an inappropriatetransfer of the focus detection frame. If objects such as human facesresembling each other cross the front of the main object, aninappropriate transfer of the focus detection frame may be caused bytemplate matching. In this case, after the state shown in FIG. 5B, thefocus detection frame is set on the face of the second object 32 andfocusing is performed.

As shown in FIG. 5C, if the user is gazing at the first object, the gazeregion marker 22 is displayed on the first object 31 and the trackingmarker 23 is displayed on the second object 32. Hence, the user canrecognize that the focusing has deviated from the first object 31. Inorder to focus on the first object 31 in the state of FIG. 5C, the usercan operate the operation button 10 b to generate a trigger as describedwith reference to FIG. 3. Similar to the example shown in FIGS. 4A to4C, the user may release S1 and press it again. As shown in FIGS. 5A to5C, if inappropriate focus tracking is caused, an opportunity for suchas readjustment of focus can be given by presenting appropriateinformation to the user.

Additionally, in the state shown in FIG. 5B, if the matching degree oftemplate matching becomes unsatisfactory due to the crossing of thefront by the object, the focus may be automatically reset at the gazeregion position. For example, if an object crossing in front issignificantly different in shape from the original template (a human'sface in the present embodiment), for example, an automobile, anappropriate corresponding region cannot be found because of the failureto transfer the focus detection frame. In such a case, it is onlynecessary to perform focusing again at the gaze region position. Thelarge difference in a template is caused until the crossing objectcompletely passes by, and as a result, the focusing is performed at thegaze region position. Subsequently, when the object passes by, thetracking of the original object starts again. In FIG. 3, in S230, it isdetermined whether or not the object has been lost as a result of nolonger finding an appropriate corresponding region. If it is determinedthat the object has been lost, focusing is performed at the gaze regionposition in S220. Consequently, even if an automobile and the likecrosses the front of a person, focusing is automatically performed againat the gaze region, and the main object is brought into focus againafter the automobile has passed by.

FIGS. 6A to 6C illustrate an example of the shot images in a case inwhich the scenes that are the same as those in FIGS. 4A to 4C are shotin the gaze region tracking mode. In FIGS. 6A to 6C, the same referencenumerals are provided for elements that are the same as those in FIGS.4A to 4C. The timing of S1 in FIGS. 6A to 6C is different from that inFIGS. 4A to 4C. FIG. 6A illustrates a state at the moment of S1. FIG. 6Band FIG. 6C illustrate a state in which S1 is maintained. Additionally,in FIGS. 6A to 6C, it is assumed that the operation mode is the gazeregion tracking mode in which focusing continues to be performed at thegaze region. That is, FIGS. 6A to 6C correspond to the case in which theprocess proceeds to from S180 to S200 in FIG. 3.

As shown in FIG. 6A, until the moment of S1, the tracking marker 23 andthe gaze region marker 22 overlap. In the gaze region tracking mode, thetracking marker and the gaze region marker overlap after the moment ofS1 as well. In the examples shown in FIGS. 6A to 6C, the thickness ofthe marker is made different in order to clearly indicate that trackingis being performed based on either the result of detection of the gazeregion or the result of detection of the object. In FIGS. 6A and 6B,since the position at which the marker is to be presented is determinedbased on the detected gaze region, the gaze region marker 22 is shown bya bold line and the tracking marker 23 is shown by a thin line. In FIG.6C, since the position at which the marker is to be presented isdetermined based on the object detected by image recognition, thetracking marker 23 is shown by a bold line and the gaze region marker 22is shown by a thin line. Note that, instead of a difference in thethickness of the marker, a difference in color may be used fordetermining the position at which the marker is to be presented byeither the result of detection of the gaze region or the result ofdetection of the object.

A case is assumed in which the gaze region can properly be detectedduring the state shown in FIG. 6A to the state shown in FIG. 6B. At thistime, the reliability of the gaze region is equal to or greater than thethreshold value and corresponds to the case in which the processproceeds from S200 to S240 in FIG. 3. At this time, the camera systemcontrol circuit 5 determines the position at which the marker is to bepresented based on the result of the detection of the gaze region. Acase is assumed in which the gaze region cannot be properly detected atsome point in time during the process proceeding the states from FIG. 6Bto FIG. 6C. The gaze region detection may not operate properly if straylight such as sunlight is present. At this time, the reliability of thegaze region is equal to or less than the threshold value, andcorresponds to the case in which the process proceeds from S200 to S250in FIG. 3. At this time, the camera system control circuit 5 determinesthe position at which the marker is to be presented based on the objectdetected by image recognition. According to the display operationdescribed with reference to FIGS. 6A to 6C, even if the output of thegaze region detection circuit is inappropriate, it is possible tomaintain in focus so as not to miss the best timing for pressing theshutter while continuing the tracking by the detection of the object. Asdescribed above, according to the image processing apparatus of thepresent embodiment, it is possible to present information to the userthat enables intuitively performing the confirmation of the focus stateand the selection of an object to be focused to the user.

OTHER EMBODIMENTS

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions (e.g., one or more programs) recorded on a storage medium(which may also be referred to more fully as a ‘non-transitorycomputer-readable storage medium’) to perform the functions of one ormore of the above-described embodiment(s) and/or that includes one ormore circuits (e.g., application specific integrated circuit (ASIC)) forperforming the functions of one or more of the above-describedembodiment(s), and by a method performed by the computer of the systemor apparatus by, for example, reading out and executing the computerexecutable instructions from the storage medium to perform the functionsof one or more of the above-described embodiment(s) and/or controllingthe one or more circuits to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or moreprocessors (e.g., central processing unit (CPU), micro processing unit(MPU)) and may include a network of separate computers or separateprocessors to read out and execute the computer executable instructions.The computer executable instructions may be provided to the computer,for example, from a network or the storage medium. The storage mediummay include, for example, one or more of a hard disk, a random-accessmemory (RAM), a read only memory (ROM), a storage of distributedcomputing systems, an optical disk (such as a compact disc (CD), digitalversatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, amemory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2017-021134, filed Feb. 8, 2017, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. An image processing apparatus, comprising:circuitry which obtains a shot image; detects a first region that is aregion gazed at by a user within a region of the shot image; determinesa second region that is a region of a main object within the region ofthe shot image; and performs control to display first region informationindicating the first region and second region information indicating thesecond region so as to be distinguishably superimposed on the shotimage; wherein, before the first instruction member is operated, thecircuitry displays the first region information and the second regioninformation based on information on the identical position, and afterthe first instruction member is operated, the circuitry displays thefirst region information and the second region information based oninformation on different positions corresponding to each region.
 2. Theimage processing apparatus according to claim 1, wherein, before thefirst instruction member is operated, the circuitry displays the firstregion information and the second region information based oninformation indicating the detected position of the first region.
 3. Theimage processing apparatus according to claim 1, wherein, after thefirst instruction member is operated, the circuitry displays the firstregion information based on the information indicating the detectedposition of the first region, and displays the second region informationbased on information indicating the determined position of the secondregion.
 4. The image processing apparatus according to claim 1, whereinthe circuitry determines the second region in accordance with anoperation of the first instruction member.
 5. The image processingapparatus according to claim 4, wherein, if an operation of a secondinstruction member that is different from the first instruction memberis detected after the first instruction member is operated, thecircuitry displays the second region information based on theinformation indicating the position of the first region.
 6. The imageprocessing apparatus according to claim 4, wherein the circuitrydisplays the second region information based on the informationindicating the position of the first region if an evaluation value withrespect to the detection of the object is lower than a threshold value.7. The image processing apparatus according to claim 1, wherein, if thefirst instruction member is operated, the circuitry determines thesecond region based on an operation mode with respect to a focusingoperation and information indicating a shooting state.
 8. The imageprocessing apparatus according to claim 7, wherein, if the operationmode is a first operation mode, the circuitry determines a regioncorresponding to the position of the region of the object determinedbased on the shot image to serve as the second region, and wherein, ifthe operation mode is a second operation mode, the circuitry determinesa region corresponding to the position of the first region to serve asthe second region.
 9. The image processing apparatus according to claim8, wherein, if the operation mode is the second operation mode thatperforms an operation of focusing on the first region and an evaluationvalue with respect to the detection of the first region is equal to orgreater than a threshold value, the circuitry determines a regioncorresponding to the position of the first region to serve as the secondregion.
 10. The image processing apparatus according to claim 8,wherein, if the operation mode is the second operation mode thatperforms the operation of focusing on the first region and theevaluation value with respect to the detection of the first region islower than the threshold value, the circuitry determines that the regionof the object determined based on the shot image to serve as the secondregion.
 11. An imaging apparatus including an imaging element thatphotoelectrically converts an object light, and an image processingapparatus, the image processing apparatus comprising: circuitry whichobtains a shot image; detects a first region that is a region gazed atby a user within a region of the shot image; determines a second regionthat is a region of a main object within the region of the shot image;and performs control to display first region information indicating thefirst region and second region information indicating the second regionso as to be distinguishably superimposed on the shot image, wherein,before the first instruction member is operated, the circuitry displaysthe first region information and the second region information based oninformation on the identical position, and after the first instructionmember is operated, the circuitry displays the first region informationand the second region information based on information on differentpositions corresponding to each region.
 12. The imaging apparatusaccording to claim 11, wherein the first instruction member is anoperation button for providing instructions for shooting preparation.13. A control method of an image processing apparatus comprising:obtaining a shot image; detecting a first region that is a region gazedat by a user within a region of the shot image; determining a secondregion that is a region of a main object within the region of the shotimage; and performing control to display first region informationindicating the first region and second region information indicating thesecond region so as to be distinguishably superimposed on the shotimage, wherein, before a first instruction member that performsinstructions for shooting preparation is operated, the first regioninformation and the second region information are displayed based oninformation on the identical position, and after the first instructionmember is operated, the first region information and the second regioninformation are displayed based on information on different positionscorresponding to each region.